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Teachers via a Multi-Methods Approach 



Background 



In its report, Who Will Teach Our Child ren? A Strategy for Improving California's Schools, the 
Califoi-nia Commission on the Teaching Profession (1985) recommended that the State have a 
credentialing process that includes three assessment components: an assessment of general 
knowledge prior to entry into a teacher education program, a test in the subject-matter field at the 
completion of the program, and a series of on-site assessments of new teachers who would be in 
a "residence" status before being fiilly credentialed. 

Current assessment requirements for teacher credentialing in CaUfomia include the California Basic 
Educational Skills Test (CBEST), the NTE Core Battery, and the NTE Specialty Area Tests. 
These tests have been reviewed by CaUfomia teachers and teacher educators in recent years in 
terms of their appropriateness for use in the credcntaling process (Wheeler and Elias, 1983; 
Wheeler, et al., 1988). The reviews were undertaken primarily because of added testing 
requirements instituted by the California Legislature, the increase in the number of credential 
candidates taking the NTE examinations in California, the increased demand for new teachers, 
and the general concern about improving teacher -cessment in subject-matter knowledge and 
teaching ability. 



California has relied on a program approval model for candidates attending in-state leacher 
education institutions (Commission on Teacher Credentialing, 1988). However, it is now generally 



accepted that: "...individual assessment provides a better way of evaluating the capabilities of 
potential teachers than does program approval" (Goertz, Ekstrom, and Coley, 1984, p.iii). The 
California Teacher Credentialing Law of 1988 calls for "Deregulation of the academic training of 
teachers by shifting the emphasis from granting credentials on the basis of program approval to 
granting credentials on the basis of comprehensive assessments of individual candidates" (Section 
44279.3,b.5). The pilot testing of existing performance and alternative assessments (Estes, et al., 
1990) and the development of prototype assessments under the California Nevy Teache .oject 
(CNTP) are two steps in mabng the change from the program approval model to the candidate 
assessment model. 

The 1987-88 review of fifteen NTE Specialty Area Tests by over 400 California educators pointed 
out some major concerns with these tests. Rrst, there is strong need to augment, or in some cases, 
replace the multiple-choice tests vnth some type of performance assessment Second, several of 
the multiple-choice tests do not match the California curriculum framevrarks as closely as the 
reviewers felt they should and therefore ne^.d to be replaced or supplemented by tests that provide 
a more comprehensive coverage of the frameworks. (Wheeler, et al., 1988) 

The California Legislature, in the California Teacher Credentialing Law of 1988 (Chapter 1355 of 
tile Statutes of 1988), provides support for developing alternative methods foi assessing beginning 
teachers, "...the Commission shall establish procedures to assess the teaching performance of 
beginning teachers on a regular basis by on-site visits of assessment teams or through the use of 
uniform exercises in a controlled setting, or both..." (Section 44279.6). As part of this effort, five 
innovative assessments were pilot-tested in the spring of 1989 at California New Teacher Project 
sites (Estes, et al., 1990). Funding was also provided for the development of five new prototype 
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assessments, including this project for development of a procedure for assessing science laboratory 
teaching performance of elementary and secondary school teachers, for pilot testing in the spring 
of 1990. 

Science as the Focus of the Assessment 

Science was chosen as the focus of this assessment for several reasons. In an era of rapidly 
developing technology and worldwide competition, the development of future mathematicians, 
scientists, and engineers remains a national priority. In order to develop top scientists, we need 
quality science teachers, starting at the elementary school level. The importance of scientific 
literacy and the need for educational reform to start with children in kindergarten are emphasized 
throughout the Project 2061 report (American Association for the Advancement of Science, 1989). 

A study of the data for sixth graders from the California Assessment Program (CAP) found that 
Wgher amounts of time allocated by schools to science instruction were associated with higher CAP 
performance levels in reading, mathematics, and writing, even for low SES schools (Wheeler, 1986- 
87). Quality science instruction provides an opportunity for students to improve skills in these 
three areas as well as to develop more advanced thinking skills (e.g., analysis, comparison, 
inference, evaluation), which are receiving increased attention of curriculum specialists (Queilmalz, 
1985) and policy makers. 

As part of their effort to improve the quality bf California's teaching force, the Commission on 
Teacher Credentialing has been reviewing the credentialing requirements. Part of this effort was 
a review of the NTE Specialty Area Tests. The tests in Biology and General Science and in 
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Chemistry, Physics, and General Science were r-A-ijwed by 34 and 25 CahTomia educators, 
respectively. The reviewers were highly trained and very experienced teachers and teacher trainers 
fiom throughout the State of California. Not only did they review a current test, item by item, and 
the test specifications, but they were also asked to comment on those aspects of the credentialing 
process for their subject area that needed to be changed. The recurring concern expressed by 
reviewers of both science tests was the need to assess laboratory demonstration and presentation 
skills and lab safetj'. Although some felt aspects 6f lab safety could be tested on a multiple-choice 
test, all agreed that laboratory presentation and demonstration skills of credential candidates could 
only be evaluated through a performance assessment or an observation of the teacher conducting 
a science laboratory activity (Wheeler, et al., 1988). 

The national need for highly trained scientists; the importance of science instruction for developing 
students' basic skills in reading, mathematics, and writing as well as more advanced thinking skills 
in these and other areas; and the concerns of many California teachers and teacher educatore who 
reviewed the NTE science tests that laboratory demonstration and presentation skills and lab safety 
be evaluated as part of the credentialing process led to this project. Having such an assessment 
for use with new teachers should address all three reasons listed above. 

Development of the As.sessment 

Several sources of information were utilized in developing the materials and procedures for this 
assessment. These include reviews of the literature on effective science teaching, other teacher 
performance assessment systems, textbooks on science teaching methodology, California's curriculum 
guides and framework for science, and California's standards for new teachers. 
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Most research on effective teaching looks at teacher-centered instruction and few studies focus 
specifically on science teaching or on student-centered teaching situations such as hands-on science 
lab activities. White and Tisher (1986), however, in their review of research on science teaching 
identified conditions that were found to contribute to a science lab. These were student 
involvement in and control over the activity, and student interaction with the teacher and with 
other students. 



A set of studies of science teachers in Australia, by Tobin and Gamett, found that science teachers 
identified by other educators as exemplary shared the following characteristics. They exhibited 
adequate pedagogical knowledge, with the secondary teachers exhibiting much stronger content 
knowledge than the elementaiy teachers. The secondary teachers were also able to explain 
scientific phenomena, demonstrate scientific principles, expand on student learning of given 
scientific concepts, and probe student understanding to diagnose partial understandings and 
misunderstandings. The major goal of the instruction of these exemplary teachers was to improve 
student understanding. They used laboratory activities and concrete models as essential means by 
which to attain this goal. Each lesson was carefully related to previously learned material and real- 
world applications. The teacher asked many questions and encouraged the students to ask 
questions. Students were provided with feedback on their progress. In addition to their 
instructional skills, these teachers had strong classroom management skills which led to high levels 
of student engagement. By moving around the room, the teachers monitored student 
understanding, engagement and behavior (Gamett and Tobin, 1988; Tobfn, 1987; Tobin and 
Gamett, 1988). 

In our review of observation assessment materials firom nine states, the following aspects of 
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effective teaching were found to be emphasized. Planning should be carried out so as to achieve 
selected objectives and take into account the individual differences of students. Instruction should 
be related to these objectives; appropriate to the content, objectives and students; and motivating. 
A variety of teaching methods should be employed to promote student engagement. Control of 
student behavior and consistency are important aspects of management. Evaluation should be used 
to monitor the effectiveness of and to improve instruction as well as to obtain information about 
the progress and needs of students. The learning environment should be positive, setting high 
expectations for students, working to improve learners' self concepts, and encouraging student 
learning and enthusiasm for learning. Communication should include feedback to students. 
Organization of time, space, materials, and equipment should provide accessibility and an attractive 
and orderly environment. The teacher's presentation of content should be accurate and relevant 
to other topics. The teacher should set gpod examples of reading, writing, and speaking skills. 

Over 100 California science educators, who were invited to serve on this Project's Science 
Assessment Development Committee or to be reviewers of this Project's materials, were asked to 
describe the purpose of laboratory work in their science classes. Their r^ponses fell into several 
major areas. A good science lab enables students to understand scientific methods and the nature 
of research and experimentation. It also provides a structure for learning sf;ience content. As 
reinforcement or an alternate approach, it increases the learning of science content taught by other 
methods. The science activity serves tc increase student interest in and improve student attitudes 
towaid science. By experiencing science, students learn to better understand and appreciate their 
environment. Another valuable benefit they listed is that laboratory skills and safety are developed 
through a science lab activity. Additionally, the lab serves to encourage higher-order thinking skills, 
promote group work and cooperation, and develop skills in other subject areas. 
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The September 1989 draft of the California Science Framework (Science Curriculum Framework 
and Criteria Committee, 1989) sets forth the following expectations for science programs in 
California's public schools: 

1. The major themes underlying science, such as stability, patterns of change, evolution, systems 
«nd interactions, scale and structure, and energy, are developed and deepened through a 
thematic approach. 

2. The three basic scientific fields of study-physical, earth, and life-are addressed, ideally each 
ysar, and the connections among them are developed. 

3. Tht character of science is fhown to be open to inquiry and controversy, and free of 
dogmatism; the curriculum promotes student understanding of how we come co know what 
we know and how we test and revise our thinking. 

4. Science is presented in connection with its applications in technology and its implications 
for society. 

5. Science is presented in connection with students' own experiences and interests, frequently 
using hands-on experiences that are integral to the instructional sequence. 

6. Students are given opportunities to construct the important ideas of science, which are then 
developed in depth, through inquiry and investigation. 

7. Instructional strategies and materials allow several levels and pathways of access so that all 
students can experience both challenge and success. 

8. Printed materials are written in an interesting and engaging narrative style; in particular, 
vocabulary is used to facilitate understanding rather than as an end in itself. 

9. Texts are not the sole source of the curriculum; ordinary materials and laboratory 
equipment, video and software, and other printed materials, such as reference books. 
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provide a substantial part of student experience. 

Assessment programs are aligned with the instructional program in both content and format; 
student performance and investigation play the same central role in assessment that they 
do in instruction. 



The California Standards of Pedagogical Knowledge and Competence for Beginning Teachers have 
criteria in the following for new teachers: Student Rapport and Classroom Environment; Curricular 
and Instructional Planning Skills; Diverse and Appropriate Teaching; Student Motivation, Conduct, 
and Involvement; Presentation Skills; Student Diagnosis, Achievement, and Evaluation; Cognitive 
Outcomes of Teaching; Affective Outcomes of Teaching; Capacity to Teach Crossculturally; 
Readiness for Diverse Responsibilities; and Professional Obligations. (Commission on Teacher 
Credentialing, 1988) 



Expertise is provided to the assessment development staff by a consultant and a committee of ten, 
all of whom teach science in public elementary or secondary schools in Northern California. They 
have several years of teaching experience and have worked with student teachers and as mentor 
teachers. The materials develope for this project were reviewed by over 60 California science 
educators in January 1990 and tried out by committee members in classrooms and with video tapes. 
Based on the comments of the reviewers and committee members, the materials were revised for 
use in the pilot testing, being conducted by Far West Lab in the spring of 1990. 



Description of the Assessment 



We identified seven Domains of teacher performance *o be covered by this assessment. They are: 
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Pedagogy, Content, Materials/Equipment, Management, Knowledge of Students, Climate, and 
Cbmmunication- Within each Domain, between two and nine Elements, or categories of 
performance, were identified. The list of Domains and Elements is shown in Table 1. 



TABLE 1 

List of Domains and Elements 

Califonih Nc# Tcichcr Projcct'i 
Sdoioe Laboratocy Asscnmait 



A. PEDAGOGY D. 
Ai* Planning 

A2. Sequence 

A3. Prelab 

A4. Directions 

A5. Explanation/Presentation 

A6. Monitoring/Adjusting 

A7* Feedback 

A8. Questioning 

A9* Closure E. 

B. CONTENT 
BL Accurate 

B2. Integrated R 
B3* Related to Objectives 

G MATERIALS/EQUIPMENT 
CI. Teacher Use 
C2. Safe Setup 

C3. Safe Practices G. 
C4. Avail >*Mty 



MANAGEMIiNT 

DJ. Grouping 

D2. Other Personnel 

D3. Routines and Transitions 

D4. Student Engagement 

D5. Timing 

D6. Student Behavior 

D7. Lab Cleanup 

KNOWLEDGE OF STUDENTS 

El. Diversity 

E2. Student Characteristics 

CLIMATE 

Fl. Interactions with Students 
FZ Interactions among Students 
F3. Attitudes 
F4. Inquiry 

COMMUNICATION 

Gl. Speaking 

G2, Writing 

G3. Listening 

G4. Strength of Presence 



For each Element, Indicators were written that describe specific teacher knowledge, skills, and 
behaviors for use in assessing performance. Student behaviors, which could reflect teacher 
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performance in certaf ,;eas, are included rs part of some Indicators. The Domains, Elements, 
and Lidicators are the heart of the assessment. They are the areas in which the observers will 
collect information for use in making judgments about performance for each of the seven Domains 
and across all Domains. 



Several approaches can be used to collect information on teacher performance. These include: 
observations, semi-structured interviews, written responses (such as essay., or open-ended test 
questions), structured simulations, and portfolios. Observations were chosen as the primary 
approach lor collecting information because they are a form of live performance testing and have 
high face validity for those concerned with ensuring that we have credentialing assessment 
requirements that are concerned with protecting students from teachers who lack minimal 
competencies and skills. However, this assessment included two additional approaches~a 
questionnaire and interviews~to increase the array of information for use in making judgments. 

There are a variety or techniques that have bean used to collect data and information when using 
classroom observations. These include checklists, rating scales, coding of behaviors and events, 
frequency counting, and scripting. For this assessment, we chose guided-note taking. This is sin:ilar 
to scripting in that the observer write out statements of behaviors and events as tney occur. 
However, the observer also writes notes about what is seen in the classroom setting, and ^^/rites 
these statements and notes in designated areas on the note-taking form during the observation. 
This approach ensures that evidence and notes are recorded for all Domains prior to the observer 
leaving the setting of the laboratory' activity aid also helps the observer sort the evidence and notes 
related to each Domain for making judgments about performance levels. 
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The observer uses four assessment components to collect information: (a) the Pre-Observation 
Questionnaire completed by the teacher, (b) the Pre-Observation Conference with the teacher, (c) 
the Observation of the teacher implementing a science laboratory activity, and (d) the Post- 
Observation Conference with the teacher. Three types of information related to the Domains, 
Elements, and Indicators are collected They include evidence, notes, and teacher responses. 
Evidence consists of descriptions and verbal statements of behaviors or events (e.g.. Teacher says 
Students pick partners for...; Teacher checks worksheet of..). Notes include additional 
information noted by the observer that wiU aid in clarifying evidence and making judgments about 
the teacher's performance in each Domain (e.g.. Teacher could be clearly heard in back of room; 
Two bottles of chemicals were not labeled). The teacher responses are the answers given by the 
teacher being assessed to the questions listed in the Pre-Observation Questionnaire and asked 
during the two conferences. 



Definition of the Science Laboratory Activity 



The focus of the assessment is a science laboratory activity. This is a student-centered, hands-on, 
inquiry-oriented activity which involves gathering data in a manner appropriate to the topic/concept 
being investigated and which can be conducted inside or outside the regular classroom or 
laboratory. A science laboratory activity requires an active role for students. Students are involved 
in conducting an experiment, ir. an investigation to gather information, or in collecting and 
analyzing data and coming to a conclusion or a better understanding of science. A lab activity 
enhances student learning by allowing students to directly observe and interpret natural or prepared 
phenomena. If it includes such steps or tasks as the showing of a video tape or film strip, ^ 
teacher demonstration, the students learning how to use equipment, a pencil-and-paper problem 
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solving session, data calculations, or the writing of lab reports, they must be in support of hands- 
on activities^ a,\;- be only a part of total lab activity. 

In order to obtain enough evidence during the observation period, the teachers have been 
instructed to plan a lab activity lasting a minimum amount of time. The times are 30 minutes for 
grades K-3, 40 minutes for grades 4-8, and 45 minutes for grades 9-12. These include those prelab 
and postlab activities which involve students. 

Selection and Training of Observers 

Observers for the pilot testing the Science Laboratory Assessment have been carefully selected. 
All have had several years experience as a teacher in California public schools, are knowledgeable 
about at least one area of science (earth, life, or physical), and have worked with student teachers, 
as a teacher trainer, or as a mentor teacher. Prior to conducting any assessments, the observers 
participated in a training program. The training covered the following: background on 
performance assessment of new teachers in California and the purpose of conducting the 
assessments; a thorough understanding of the Domains, Elements, and Indicators; familiarity with 
the assessment components and materials; an understanding of the procedures and awareness of 
the guidelines for conducting each component; a discussion of simulated responses, evidence, and 
notes; practice in conducting conferences through role playing, and in doing observations and 
collecting evidence using video tapes; an unJerstanding of the process for combining the evidence, 
notes, and responses and using them to make judgments about performance; and a knowledge of 
the procedures to be followed throughout the assessment process. 
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The training for the spring 1990 pilot testing lasted three days. On the first day, the trainees 
worked on their own, reading the Observer's Handboolc. and thoroughly familiarizing themselves 
with the Domains, Elements, and Indicators, and with the assessment materials, forms, and 
procedures. 

The trainees then met as a group for two days. The trainers reviewed the areas covered on the 
first day and answered questions the trainees had. Then trainees were provided with a set of 
simulated responses, evidence, and notes, and discuss the format and the nature of the information 
collected, its relevance to the Domains and Elements, and the judgments and summary remarks 
made based on the information. 

Next trainees were paired to role play the conducting of a conference. Trainers monitored this 
activity and assisted them in developing prompts, acairately recording responses, and following 
the guidelines and procedures in the Observer's Handbook for conducting conferences. Each 
trainee played the roles of observer and of teacher being assessed. 

When trainees completed the activities related to the conferences, they reviewed, as a group, the 
procedures and guidelines for doing the observation of the sciences laboratory activity. They then 
reviewed segments of video tapes and related documents (e.g., questionnaire, conference notes) for 
different grade levels. Th^ concentrated on collecting evidence and notes for two to three 
Domains at a time (Pedagpgy and Content; Materials/Equipment and Management; Knowledge of 
Students, Climate, and Communication). The video tape segments included examples of both 
acceptable and unacceptable performanra levels. At the end of each segment, the trainees share 
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the information they had collected and discussed how they would judge that teacher's performance 
level, based on the video tape and related documents. 

Tteinees were then provided with a QuestioLnaire, and responses to the Pre-Observatioa 
Conference, corresponding to a video tape for a fuU science lab activity. They reviewed the 
responses to these documents and then observed the video tape activity in its entirely, taking notes 
for all seven Domains. When they finished viewing this video tape, they were given the responses 
for the Post-Observation Conference. TTie trainees linked the evidence, notes, and responses to 
each Element; entered the information on the Document Sorting Form; and made judgments 
about performance at the Domain level (2 if minimally acceptable for credentialing or better, 1 if 
not acceptable, and X if cannot judge because of lack of evidence). They identified those Domains 
and Elements for which they lacked sufficient information and discussed ways they might obtain 
it (e.g., examples of "Other Questions" for tac Post-Observation Conference). The trainees shared 
and discussed their evidence, remarks, and judgments until it was clear that they could record 
evidence properly, develop suitable remarks, and make reasonable Domain judgments. They 
repeated this process for a video tape of an activity at another grade level. The trainers reviewed 
each trainee's completed forms for these video tapes. Trainers assessed each trainee on properly 
conducting a conference, asking prompts and recording responses; recording evidence and notes 
properly for the observation; sorting the information by Elements in a reasonable manner, making 
sound remarks and reasonable judgments based on the information for the Document Sorting 
Form; and filling in the forms in a clear and complete manner. 



ERIC 



16 



16 



Summary 



Performance assessments extend our ability to measure the complexities of teaching. Although they 
are far more costly to administer than the currently prevalent multiple-choice tests, they have many 
more potential benefits. Multiple-choice tests allow us to sample a wide array of knowledge and 
skills. Performance assessments permit in-depth assessment, can cover the ability to apply 
knowledge and skills, and seem to better reflect authentic teaching behaviors. They can be used 
not only for assessing prospective teachers for credentialing and certification; they can also be used 
in pre-service training, self-assessment, staff development, and teacher evaluation and recognition. 

This assessment, and many others, are still in an early stage of development. However, this is an 
exciting time for exploring alternatives to traditional forms of assessments. Hopefully this move 
will help attract the strongest candidates to the teaching profession and will improve the 
preparation, support, development, and retention of teachers in all subject areas. 
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